# Copying sparse files using Finder, gcp or ginstall may result in corrupted copies.
# While reading the entire file content returns valid data, scanning for allocated
# segments may return holes where valid data is actually written,
# hence a sparse copy will not include these segments and includes zeroes instead.

# Link to materials provided as a PoC:
# https://httpstorm.com/apple/2021-10-07/
# cc1.7z                  includes original and corrupted file
# steps to reproduce.txt  this file / e-mail
# test.sparseimage.7z     the build image where I reproduced the issue.
# Note that after dismounting the image and mounting it again, the sparse files
# inside must be recreated in order for the sparse copies to get corrupted.

# Steps to reproduce:
# Test machine: MacBook Pro 16", 2TB SSD, 64 GB RAM.
# macOS 11.6 is installed on a case sensitive encrypted APFS volume.
# Xcode is installed.

# Setup OpenWRT build environment, and compile a test image:
# Open Terminal, and enter the following commands:
xcode-select --install

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

brew install apr apr-util arping asciidoc autoconf automake bash bdw-gc berkeley-db binutils binwalk boost brotli bzip2 c-ares cmake coreutils curl dash diffutils docbook e2fsprogs fastjar findutils flex freetype gawk gcc gdbm gettext git git-extras gmp gnu-getopt gnu-sed gnu-tar gnutls gputils grep guile icu4c intltool isl jemalloc libev libevent libffi libgcrypt libgpg-error libidn2 libmetalink libmpc libnet libpng libssh2 libtasn1 libtool libunistring libxml2 libxslt libyaml lz4 m4 make mercurial mpdecimal mpfr mtr ncurses nettle nghttp2 numpy openblas openldap openssl@1.1 p11-kit p7zip patchutils pcre pcre2 perl pkg-config python@3.8 python@3.9 quilt readline rtmpdump ruby source-highlight sqlite ssdeep subversion tcl-tk unbound unzip utf8proc wget xz zlib zstd

cd /usr/local/bin
ln -s /usr/local/opt/make/libexec/gnubin/make make

# Edit /etc/shells:
sudo nano /etc/shells

# Add the following line at the end:
/usr/local/bin/bash

# Change the default shell to bash, and exit
chsh -s /usr/local/bin/bash
exit

# open Disk Utility
# press Command-N to create a new virtual disk
# Image Format: sparse disk image
# save as: ~/vhd/test
# Name: test
# Size: 50 GB
# Format: APFS (Case-sensitive)
# Encryption: none
# Partitions: Single partition - GUID Partition Map
# Save
# open a new terminal

git clone https://github.com/openwrt/openwrt.git
cd openwrt

./scripts/feeds update -a
./scripts/feeds install -a

make defconfig
make menuconfig

# Note: for detailed log use
# make V=sc

make -j 16

 make[1] world
 make[2] tools/compile
 make[2] package/cleanup
 make[3] -C tools/flock compile
 make[3] -C tools/xz compile
 make[3] -C tools/sed compile
 make[3] -C tools/patch compile
 make[3] -C tools/tar compile
 make[3] -C tools/cpio compile
 make[3] -C tools/autoconf-archive compile
 make[3] -C tools/m4 compile
 make[3] -C tools/ninja compile
 make[3] -C tools/lzma compile
 make[3] -C tools/mtools compile
 make[3] -C tools/patch-image compile
 make[3] -C tools/coreutils compile
 make[3] -C tools/sstrip compile
 make[3] -C tools/xxd compile
 make[3] -C tools/zip compile
 make[3] -C tools/expat compile
 make[3] -C tools/meson compile
 make[3] -C tools/pkgconf compile
 make[3] -C tools/libressl compile
 make[3] -C tools/autoconf compile
 make[3] -C tools/automake compile
 make[3] -C tools/missing-macros compile
 make[3] -C tools/cmake compile
 make[3] -C tools/mkimage compile
 make[3] -C tools/dosfstools compile
 make[3] -C tools/libtool compile
 make[3] -C tools/e2fsprogs compile
 make[3] -C tools/flex compile
 make[3] -C tools/mklibs compile
 make[3] -C tools/fakeroot compile
 make[3] -C tools/gengetopt compile
 make[3] -C tools/patchelf compile
 make[3] -C tools/gmp compile
 make[3] -C tools/bison compile
 make[3] -C tools/zstd compile
 make[3] -C tools/zlib compile
 make[3] -C tools/mpfr compile
 make[3] -C tools/firmware-utils compile
 make[3] -C tools/lzma-old compile
 make[3] -C tools/squashfskit4 compile
 make[3] -C tools/mtd-utils compile
 make[3] -C tools/make-ext4fs compile
 make[3] -C tools/findutils compile
 make[3] -C tools/bc compile
 make[3] -C tools/squashfs compile
 make[3] -C tools/mpc compile
 make[3] -C tools/quilt compile
 make[3] -C tools/padjffs2 compile
 make[2] toolchain/compile
 make[3] -C toolchain/binutils compile
 make[3] -C toolchain/gdb compile
 make[3] -C toolchain/fortify-headers compile
 make[3] -C toolchain/gcc/initial compile
 make[3] -C toolchain/kernel-headers compile
 make[3] -C toolchain/musl compile
    ERROR: toolchain/musl failed to build.
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
make: *** [/Volumes/test/openwrt/include/toplevel.mk:230: world] Error 1

# We expect the build process to fail here, because of this bug.
# At the end of make[3] -C toolchain/gcc/initial compile
# /Volumes/test/openwrt/build_dir/toolchain-arm_cortex-a9+vfpv3-d16_gcc-11.2.0_musl_eabi/gcc-11.2.0-initial/gcc/cc1
# and some other files are installed to
# /Volumes/test/openwrt/staging_dir/toolchain-mips_24kc_gcc-11.2.0_musl/libexec/gcc/mips-openwrt-linux-musl/11.2.0/
# using the following command, note that -c is ignored
# ginstall -c $src $dst
#
# ginstall and gcp are part of coreutils. The latest version 9.0 adds support
# for copying sparse files, by only copying the allocated segments. Finder does
# the same, so copies of the aforementioned cc1 made using Finder will also be
# broken and will match exactly the hash of the copies made using ginstall or gcp.
# Comparing original vs broken copy shows that some segments are replaced with
# zeroes. Copies made using /bin/cp are accurate, because the built-in cp does
# not support sparse files and copies the entire content. Downgrading coreutils
# to version 8.32 also results in accurate copies for the same reason. We can
# disable support for sparse files in coreutils 9.0, by building the latest
# master with the following change:

# Workaround: disable SEEK_HOLE in coreutils, src/copy.c, line 1098
--- #ifdef SEEK_HOLE
+++ #if defined(SEEK_HOLE) && !defined(__APPLE__)

# Then we replace ginstall with the one that does not support sparse files,
# clean and rebuild the broken gcc-initial and resume the build:
make toolchain/gcc/initial/{clean,compile} -j 16
make -j 16

# The build will succeed.
# If we try to copy cc1 or some of the other sparce files from
# /Volumes/test/openwrt/build_dir/toolchain-arm_cortex-a9+vfpv3-d16_gcc-11.2.0_musl_eabi/gcc-11.2.0-initial/gcc/
# using Finder, gcp or ginstall (with support for sparse files), the resulting
# copies will be broken. However if we dismount test.sparseimage, the next time
# we mount it and make copies, the copies will be accurate.
# Rebuild gcc-initial again:
make toolchain/gcc/initial/{clean,compile} -j 16

# If we make copies using Finder, gcp or ginstall (with support for sparse files),
# the resulting copies are broken again.